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Confirmatory Factor Analyses of Multitrait-multimethod Data: 
Many Problems and a Few Solutions 



ABSTRACT 

During the last 15 years there has been a steady increase in the 
popularity and sophistication of the confirmatory factor analysis (CFA) 
approach to multit- «t-multi method <MTH1) data. There exist, however, 
important problems, the most serious being the ill-defined solutions that 
plague MTMM studies and the assumption that so-called method factors 
reflect primarily the influence of method effects. In three different MTMM 
studies ill-defined solutions were frequent and alternative 
parameterizations designed to solve this problem tended to mask the 
symptoms instead of eliminating the problem. More importantly so-called 
method factors apparently represented trait variance in addition to, or 
instead of, method variance for at least some models in all three studies. 
Further support for this counter interpretation of method factors was 
found when external validity criteria were added to the MTMM models and 
correlated with the trait and so-called method factors. This problem, when 
it exists, invalidates the traditional interpretation of trait and method 
factors and the comparison of different MTMM models. A new specification 
of method effects as correlated uniquenesses instead of method factors was 
less prone to ill-defined solutions and, apparently, to the confounding of 
trait and method effects. 
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Multitrait-multi method data 1 

The purpose of this investigation is to demonstrate and critically 
evaluate recently developed applications of confirmatory factor analysis 
<CFA> to eultitrait-eultieethod (MTMM) data. Caepbell and Fiske (1959) 
argued that construct validation requires Multiple indicators of the same 
construct to be substantially correlated with each other but substantially 
less correlated Mith indicators of different constructs. They proposed the 
MTMM design in which each of a set of Multiple traits is assessed with each 
of a set of multiple Methods of assessment, and developed four g, idelines 
for evaluating MTMM data. Their HTMM design has become, perhaps, the Most 
frequently employed construct validation design, and their original 
guidelines continue to be the most frequently used guidelines for examining 
MTMM data. iMportant problems with their guidelines are, however, well 
known (e.g., Althauser t, Heberlein, 1970; Alwin, 1974; Campbell fc 
O'Connell, 1967; Marsh, in press; Hothke, 1984) and have led to Many 
alternative analytic approaches (e.g., Browne, 1984; Hubert * Baker, 1978; 
Jackson, 1969; 1977; Marsh, in press; Marsh & Hocevar, 1983; Schmitt, 
Coyle, i, Saari, 1977; Schmitt t, Stults, 1986; Stanley, 1961; Hothke, 1984). 
Factor analytic approaches (e.g., Boruch & Hoi 1 ins, 1970; Joreskog, 1974; 
Marsh, in press; Harsh t, Hocevar, 1983; Mi daman, 1985) or mathematically 
similar path-analytic approaches (e.g., Herts t, Linn, 1970; Scheitt, Coyle 
t Saari, 1977) currently appear to be the most popular approach and will be 
the focus of the present investigation. 

A GENERAL MTMM MODEL AMD A TAXONOMY OF ALTERNATIVE MODELS 
IbS gSQfiCsi MTMM Model, 
In the CFA approach to MTMM data a priori factors defined by different 
Measures of the same trait support the construct validity of the measures 
but a priori factors defined by different traits measured with the same 
-ethod argue for Method effects. For purposes of the present investigation 
I will emphasize a general MTMM Model (Table 1) adapted from Joreskog 
<1974, also see Marsh & Hocevar, 1983, HidaMan, 1985) in which: a) there 
are at least 3 traits (T-3) and 3 Methods (M-3), b) T x M Measured 
variable, are us* to infer T ♦ M a priori co~on factors, c) each Measured 
variable loads on the one trait factor and the on. Method factor that it 
repress but is constrain* so as not co load on any other factors, d) 
correlations among the trait factors and among the Method factors are 
freely estimated, but correlation, between trait and method factors are 
constrained to be zero. For this model I assume there are at least three 
trait, and three methods, but alternatives have been proposed fw studies 
with only two methods (Kenny, 1979) or only two traits (Marsh « Hocevar, 
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Multitrait-multimethod data 2 
1983). While some research.™ have estimated correlations between trait 
and eethod factors there are ieportant logical, interpretive, and pragmatic 
reasons for fixing these correlations to be zero (see Jackson, 1974; Marsh 
* Hocevar, 1983, Hidaman, 1985). This constraint allocs the deposition 
of variance into additive trait, eethod, and error components, and without 
this constraint the solution is aleost always empirically underidentif ied 
(also see Hidaman, 1983, Hothke, 1984). In justifying this constraint, 
Joreskog (1971, p. 128) noted that: "This is our way of defining each 
eethod factor to be independent of the particular traits that the eethod is 
used to measure. In other words, eethod factors are what is left over after 
all trait factors have been eliminated." 

Insert Table 1 About Here 



In the present investigation CFA models werefit with LISREL V (Joreskog 
I Soi boa, 1981) and three design matrices from LISREL Mere used to define all 
the HTMM models. For T=3 traits and H=3 methods (see Table 1) the three 
design matrices are: a) Lambda Y, a 9 (M x T = number of measured variables) 
by 6 (H ♦ T * number of factors) matrix of factor leadings, Psi, a 6 (H ♦ T » 
number of factors) x 6 factor variance-covariance matrix of relations among 
the factors, and c) Theta, a 9 (M x T = number of measured variables) x 9 
matrix of error /uniquenesses in which the diagonal values are analogous to 
one minus the communality estimates in exploratory factor analyses. All 
parameters (Table 1) with values of 0 or 1 are fixed and values for other 
parameters are estimated so as to maximize goodness of fit. Standard errors 
are estimated for all estimated parameters but not for parameters with fixed 
values. This model is easily modified to accommodate more traits or methods, 
to conform to other models and other parameter iz at ions that will be 
described, or to incorporate unique factors for the measured variables 
(Rindskopf, 1983). 

£ lexongmy. of Alternative novels 
Researchers have proposed many variations of the general HTMM model to 
examine inferences about trait or method variance or to test substantive 
issues specific to a particular study (e.g., Bagozzi, 1978, Joreskog, 1974, 
Harsh, Barnes * Hocevar, 1983, Marsh & Hocevar, 1983, Hi daman, 1985, Hothke, 
1984). Mi daman (1985) proposed an important taxonomy of such models that 
systematically varied different characteristics of the trait and method 
factors. This taxonomy was designed to be appropriate for all HTMM studies, 
to provide a general framework for making inferences about the effects of 
trait and method factors, and to objectify the complicated task of 
formulating models and representing the HTMM data. One purpose of the present 
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investigation is to evaluate ths taxonomy in relation to these goals and to 
describe an expansion of the taxonoay formulated for the present 
investigation. The expanded taxonomy (see Table 2) represents all possible 
combinations of 4 trait structures <trait structures 1 - 4) and 5 Method 
structures (eethod structures A - E). The 4 trait structures posit no trait 
factors (1), one general trait factor defined by all Measured variables (2), 
T uncorrelated trait factors (3), and T correlated trait factors (4). The 3 
Method structure* posit no Method factors (A), one general Method factor 
defined by aH Measured variables (B), M uncorrelated Method variables (C), H 
correlated Method factors (D>, and eethod effects inferred on the basis of 
correlated uniqueness (E). This taxonoay differs fro* Widaean's original 
taxonoay only in the addition of Method structure E. 

Insert Table 2 About Here 
The general factors posited in Method structure B and trait structure 2 
eay present interpretive or estimation probleMs. WidaMan (1985) avoided soae 
probleas by constraining each general factor to be uncorrelated with all 
other factors and this constraint is used here. The rationale for this 
constraint is consistent with the requirement that trait and Method factors 
be uncorrelated. Models IB and 2A are, however, are the same, whereas Model 
2B requires one additional, perhaps arbitrary, zero constraint to assure 
rotational identification. Finally, even for Models that contain a general 
eethod factor in coMbination with T trait factors, or a general trait factor 
in coMbi nation with M Method factors, the interpretation of the general 
factor May be problematic. 

POTENTIAL PROBLEMS IN THE ESTIMATION AND INTERPRETATION OF MTMM MODELS 

Ggodnesf Of Fit 

An important, unresolved problem in CFA is the assessment of goodness of 
fit. To the extent that a hypothesized Model is identified and is able to fit 
the observed data, there is support for the model. The problem of goodness of 
fit is how to decide whether the predicted and observed results are 
sufficiently alike to warrant support of a model. Mhereas t vilues can be 
used to test whether these differences are statistically significant, there 
is a growing recognition of the inappropriateness of this classical 
hypothesis testing approach. Because hypothesized models are only designed to 
approximate reality, all such restrictive models are a priori false and will 
be shown to be false with a sufficiently large sample size (Cudeck & Browne, 
1983! Harsh, Bella fc McDonald, in press} McDonald, 1983). Hence, a variety of 
fit indices have been derived to aid in this decision process such as the 
X /of ratio and the Tucker-Lewis index (TLI| Tucker fc Lewis, 1973) that are 
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uMd her*. In simulation studies erf more than 30 such indices Harsh, Balla 
and McDonald (1988) and Marsh, McDonald and Balla (1987) found that both the 
X /df and TLI indices imposed apparently appropriate penalty functions for 
the inclusion of additional parameters that controlled for capitalizing on 
chance, whereas the TLI Mas the only widely used index that was also 
relatively independent of sample size. The TLI is emphasized in subsequent 
discussions, but values for other fit indices like the Bentler and Bonett's 
(1980) index can easily be computed from the results. 

Model selection must be based on subjective evaluation of substantive 
issues, inspection of parameter values, model parsimony, and a comparison of 
the performances uf competing models as well as goodness of fit. In the 
application of CFA to MTMM data there is an unfortunate tendency to under- 
emphasize the examination of parameter estimates and to over -emphasize 
goodness of fit. If a solution is ill-defined, then further interpretations 
must be made very cautiously if at ail. If the parameter estimates for a 
model make no sense in relation to the substantive, a priori model, then fit 
may be irrelevant. 

As described by Bentler and Bonett (1980), when two models are nested 
the statistical significance of the difference in the *s can be tested 
relative to the difference in their df . Hi daman (1985) emphasized this 
feature in developing his taxonomy of MTMM models and in comparing the fit of 
different models. However, the problems associated with the application of 
the classical hypothesis testing approach also apply to this test of X 
differences. When the sample size is sufficiently large the saturated model 
(i.e., a model with df « 0) will perform significantly better than any 
restricted model (see Cudeck l> Browne, 1983) such as those in Table 2, thus 
making problematic the interpretation of tests between any two restricted 
models. Furthermore, many important comparisons are not nested and so cannot 
be made with this procedure (e.g., the trait-only (4A) and method-only (ID) 
models in Table 2). Because of these problems with the \ difference test, a 
perhaps more useful test is simply to compare the TLIs for competing models. 

Insert Table 2 About Here 
PfiQClv. Defined Sglytipns 

Poorly defined solutions represent a serious, unresolved problem for CFA 
that is particularly prevalent in MTMM studies. Poorly defined solutions 
refer to underidentif ied or empirically under i dent if ied models (Kenny, 1979; 
Hothke, 1984), failures in the convergence of the iterative procedure used to 
estimate parameters, parameter estimates that are outside their permissible 
range of values (e.g., negative variance estimates called Heywood cases), or 
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standard error* of parameter estimates that are excessively large. Each of 
these problems is an indication that the empirical solution is poorly 
defined, even if the model is apparently identified otherwise and even if 
goodness of fit is adequate (Joreskog * Sorbom, 1991). Such problems are 
apparently more likely when: the sample size is small; there are few 
indicators of each latent factor; measured variables are allowed to load on 
•ore than one factor; "measured variables are highly correlated; there is a lot 
of missing data and covariance matrices are estimated with pair hi se deletion 
for missing data; and the model is misspecif ied. Knowingly or unknowingly such 
problems are usually ignored, and the implications of this practice have not 
been explored for MTMM studies. Although there is no generally appropriate 
resolution for such problems, alternative parameter iz at ions of the MTMM model 
(see below) may eliminate some improper parameter estimates. 

There are apparent ambiguities about the identification status of MTMM 
models. Some researchers (e.g., Alwin, 1974; Browne, 1984; Joreskog, 1974; 
Schmitt, 1978) suggest that models with correlations between traits and 
methods are permissible, and Long (1983, p. 55) claimed to prove the 
identification for this model. However, Boll en and Joreskog (1985) 
demonstrated that the criteria used by Long were not sufficient to demonstrate 
identification, and Hidaman (1985, p. 7) explicitly eliminated such models 
from his taxonomy, claiming that they "are very likely not identified." 

In order to test the identification status of a model with correlated 
traits and methods David Kenny (personal communication, 23 January, 1987) 
used simulated data "to see if LISREL could recover loadings for your model 
40 with traits and methods correlated. It did so, but not exactly. It was not 
clear whether the difference was due to under-identif i cat ion or rounding 
error." I also attempted to fit model 40 with correlations between method 
and trait factors to the simulated population covariance matrix published by 
Cole and Maxwell (1985) in which the population correlations between trait 
and method factors were simulated to be zero. Mhereas I was able to recover 
the population values, it took more than 500 iterations. For their sample 
■atrices that included random error, however, the solutions failed to 
converge after more than 1000 iteration,. It appear, that whereas the model 
with correlated trait and method factors may technically be identified, it is 
unlikely to result in a proper solution for actual data so that it is of 
little practical use. Because models in Table 2 do not posit traits/method 
correlations and because all studies considered here have at least 3 traits 
and 3 methods, these ambiguities will not be examined here but they 
illustrate that the issue of identification has not been resolved. 



ERIC 



8 



Multitrait-multi method data 6 
CilftCElt PKMtiirtiaUsQSi PfitfintUl Cures For Poorly Defined Solutions 

IbS 5UQd§rg; Eara f eJ:erization§ i In order for the models in Table 2~to be 
identified, one parameter for each latent factor must be fixed -- typically at 
a value of 1.0 (see Joreskog & Sorbom, 1981; Long, 1983). This is usually 
done by: (a) fixing the factor loading of one measured variable for each 
latent factor to be 1.0 and estimating the factor variance, or (b) fixing the 
factor variance of eadh latent factor to be 1.0 (so that the factor 
variance/covariance matrix is a correlation matrix) and estimating all the 
factor loadings. For purposes of the present investigation these Mill be 
called the fixed factor loading and fixed factor variance parameterizations, 
and collectively they Mill be referred to as the standard parameterizations. 
So long as the CFA solution is Mell defined both parameterizations are 
equivalent, but fixing the factor variances introduces an implicit inequality 
constraint that restricts the factor variances to be nonnegative. Thus, fixing 
factor variance estimates may lead to a proper solution when fixing factor 
loadings does not. 

IbS Rindskgof Barameteri£ation x Rindskopf (1983) proposed a solution for 
negative uniqueness estimates by using M x T additional factors — one unique 
f actor for each of the M x T measured variables — to define each 
uniquenesses. Because the factor loading on each unique factor is the square 
root of the uniqueness, the uniqueness is implicitly constrained to be non- 
negative. Joreskog (1981), commenting on the merits of imposing inequality 
constraints, noted that if a solution is inadmissible, then LISREL Mill find a 
solution outside the permissible parameter space whereas the imposition of 
inequality constraints Mill produce a solution on the boundary of the 
parameter space. Joreskog (p. 91) concluded: -In both cases the conclusion 
Mill be that the model is Mrong or that the sample size is too small." 
Similarly, Dillon, Kumar and Mulani (1987) noted that in their research the 
Rindskopf parameterization always resulted in the offending parameter estimate 
taking on a zero value that resulted in the same solution as simply fixing the 
parameter to be zero. 

ft&bQd Stcygture. § — 50 2itern.ative cgngeetual izatipn of method 
variiQMi Method variance is an undesirable source of systematic variance that 
distorts correlations between different traits measured with the same method. 
As typically depicted in MTMM models (i.e., method structures C and D) a 
•ingle method factor is used to represent the method effect associated with 
variables assessed by the same method. The effects of a particular method of 
assessment are implicitly assumed to be tridimensional and the sizes of the 
method factor loadings provide an estimate of its influence on each measured 



ERIC 



9 



Multitrait-multimethoo data 7 

variabl.. toe, method structures C and 0 restrict method covariance 
components to have a congeneric-like structure (but see Wothke, 1984). 
Alternatively, method effects can be represented as correlated uniquenesses 
(method structure E) and this representation does not assume either the 
'^dimensionality of effects associated with a particular method or a 
congeneric structure. Kenny (1979; also see Harsh & Hocevar, 1983; Marsh, in 
press) proposed this method structure for the special case in which there are 
only two traits, but it is also reasonable when there are more than two 
traits. Method structure E also resembles McDonald's multi-mode analysis 
(1985) and Browne's multiple battery analysis (1980). 

Method structure E corresponds most closely to method structure C (Table 
2) in that the method effects associated with one method are assumed to be 
uncorrected with those of other methods. When there are 3 traits and the 
solutions are well-defined, method structures C and E are merely alternative 
parameter izat ions of the same model. When T > 3, however, the number of 
correlated uniquenesses in method structure E (M x (T x (T-D/2) is greater 
than the number of factor loadings used to define method factors in method 
structure C (T x M) . Thus method structure C is a special case of method 
structure E in which each method factor is required to be uni dimensional and 
this assumption is testable when T > 3. 

A particularly important advantage of method structure E is that it 
apparently eliminates some improper solutions without limiting the solution 
space or forcing parameter estimates to the boundaries of the penussible 
space. Because method variance is one source of uniqueness, uniqueness is 
reflected in both method factors and error /uniquenesses. Improper solutions 
are frequently due to either negative method factor variances or negative 
error /uniquenesses, but not both. In method structure E all sources of 
uniqueness are contained in the diagonal of Theta, and in many cases ~ as 
demonstrated in the present investigation - this combined influence will not 
be negative even when method factor variances or uniquenesses are negative for 
other parameterizations. Thus, even when there are 3 traits so that method 
structures C and E are equivalent when the solutions are well defined, it is 
possible that method structure C will results in poorly defined solutions 
Hherea, method structure E will not. Hhen there are more than three traits it 
is possible for method structure E to fit the data better than either method 
structures C or D, thus calling into question the assumed unidimensionality of 
method effects in structures C and D. 

ergbiffj i Q &b t iQitrBMi.tiQQ fff Jra.a sQd. Mjjbad Factors 
Midaman's taxonomy and the MTMM models in Table 2 implicitly assume that! 
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a) method factors represent method variance, b) trait factor, present trait 
variance, c) a general factor in combination with trait factors represents 
method variance, and d) a general factor in combination with method factors 
present, trait variance. For present purposes these assumptions Kill be 
referred to as the traditional interpretation of the MTMM models. These 
assumptions are probably reasonable when correlations among the trait factors 
and among the method factors are small, but this situation is unusual. These 
assumptions may not be reasonable when correlations among trait factors and 
correlations among method factors are substantial. For present purposes I 
Mill examine the possibility that so-called method factors actually reflect 
trait variance, but the problem might also apply to so-called trait factors 
that actually reflect method variance. 

In most hTMH studies the multiple traits are correlated and this may 
produce a general trait factor that makes ambiguous the interpretation of so- 
called general method factors or even correlated method factors. When traits 
are substantially correlated, the so-called general method factor (method 
structure B) may represent trait variance instead of, or in addition to, 
method variance. When traits are substantially correlated, each so-called 
correlated method factors (method structure D) may represent this general 
trait factor and correlations among the method factors may represent the 
convergence of this general trait across the methods of ass'jssment. If this 
problem exists, the traditional interpretation of MTMM models and the 
comparison of alternative models is unjustified. Hence, tests of this 
plausible counter interpretation of method factors must be examined. 

Results to be discussed here suggest that the traditional interpretation 
of method factors may be unjustified if: (a) interpretations based on the 
Campbell -Fiske guidelines and an examination of the MTMM matrix differ 
substantially from those based cn the CFA approach (there are, of course, 
problems with the Campbel 1 -Fi ske approach, but if both the Campbell -Fiske and 
the CFA approaches lead to consistent conclusions then confidence in these 
conclusions is increased); (b) substantive theory dictates an expected pattern 
of correlations among trait factors that is not supported; (c) the substantive 
nature of the data dictates an expected pattern of correlations among method 
factor that is not supported (though a priori hypotheses of relations among 
method factors may be difficult to formulate); (d) Model 4A (trait factors 
only) and ID (method factors only) both fit the data reasonably well and Model 
40 provides only a modest improvement; (e) the amount of variance explained by 
trait factors is substantially reduced by the inclusion of method factors; (f) 
external validity criteria collected in addition to the MTMM variables are 
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account of the theoretical rationale, the i2 MTMH variables, and the external 
validity cntenon. My application of the Campbell-Fiske guideline, suggested 
strong support for convergent validity. However, support for discriminant 

va idity .as problematic^nd t ppeared to be method variance associated 

.Ih at least M2 and M4. The substantive nature of the data indicates that 
he tra s should be substantially correlated, but there is no a priori basis 
for porting the relative size of these different correlations. Finally, f" 
models in which the external validity criterion was added, the criterion 
should be: a) more correlated with specific and general trait factors than 
with specific and general met hod factors; and b) most highly correlated with 
the behavioral trait component. 

CFA models similar to those considered here have been applied to this 
data by Bagozzi (1978), Schmitt (1978), and Widaman (1985). Schmitt (1978) 
excluded one of the methods and estimated trait/method correlations, and so 
his results are not comparable. Bagozzi (1978) fit Model 4A to the 12 
variables considered here, but an inspection of correlations between the 
error/uniquenesses led him to eliminate one of the methods from subsequent 
analyses. It should be noted that such correlated uniquenesses are indicative 
of a method effect as depicted in method structure E. Widaman (1985) also 
noted this apparent misinterpretation of method effects and was critical of 
other conclusions by Bagozzi. Widaman (1985) fit many of the models used here 
and chose to represent the MTMM data with Model 4D. However, his solution for 
Model 4D was poorly defined in that an err or /uniqueness was estimated to be 
zero and had a large standard error . None of these previous CPAs of the 
Ostrom data incorporated the external validity criterion included here, 
iibayior of the Solutions For Different Pa ramete rizing, 

All models in Table 2 were tested with both the fixed factor loading and 
the fixed factor variance parameter izat ions, and the Sindskopf 
parameterization was used when both standard parameterization, produced poorly 
defined solutions (Table 3). For the fixed factor loading parameterization, 7 
of the 19 models were poorly defined as indicated by a failure to converge or 
improper solutions. For the fixed factor variance parameterization, 5 of these 
7 models were still poorly defined but the problem symptoms were not always 
the same. When these five models were tested with the Rindskopf 
p-ameterization, one solution was improper and the remaining four had 
error/uniquenesses estimates close to zero with extremely large standard 
errors. In Model ID there were factor correlations greater than 1.0 for all 
three parameterizations, demonstrating that none of the parameterization, 
protect against this type of improper solution. Whereas the different 
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parameterization varied in their behavior and manifest symptoms, none 
eliminated the poorly defined solutions. 
Method Structure E 

In method structure E correlated uniquenesses are used to represent 
method effects, and in support of this structure all four solutions based on 
it are well defined. When there are 3 traits, method structure E is equivalent 
to method structure C -so long as the models are well defined. For the Ostrom 
data this was demonstrated for Models 1C and IE, but Models 2C, 3C, and 4C 
were poorly defined for all three parameter izations. Even though Model 2C 
failed to converge for either of the standard parameterizations, the parameter 
estimates for the trait factors and overall fit were nearly the same as for 
Model 2E. Model 3C converged to an improper solution for the fixed factor 
variance parameterization, but the parameter estimates for trait factors and 
overall fit were the same as for Model 3E. Model 4C converged to an improper 
solution for the fixed factor loading parameterization but parameter estimates 
for trait factors and overall fit were the same as for Model 4E. The Rindskopf 
parameterization eliminated improper solutions for Models 2C, 3C, and 4C, but 
resulted in err or /uniqueness estimates of zero with large standard errors. 
These findings suggest that method structure E is a better representation of 
method effects than method structure C. 

It»§ substantive interpretation of trait and method, factors 

Interpretations based on a ust the MTMM Data,. Model 4D provides an 
exceptionally ,ood fit to the data, but there are problems with the solution. 
First, it is poorly defined for all three parameterizations. Second, the trait 
factors are very weak in that 7 of the 12 factor loadings are not 
statistically significant, and this contradicts conclusions based on the 
Campbell -Fiske guidelines. Models 4C, 4E, and 3D also fit the data very well 
(TLIs > .98), and Models 4A, ID and even 2A/1B explain most of the variance 
(TLIs > .9). In contrast to Model 4D, Models 4E, 4C, and 4A have strong trait 
factors for which all factor loadings are significant. As noted by Widaman it 
may be problematic to compare trait and method variance for this data because 
most of the variance can be explained by either trait or method factors and 
neither trait nor method factors uniquely explain much variance. Also, since 
trait factor loadings are so much lower when correlated method factors are 
included, these so-called method factors may reflect trait variance. 

Widaman (1985) chose Model 4D to represent this MTMM data on the basis of 
fit. However, Model 4E (Tat'e 4) also provides a good fit and has important 
advantages over Model 4D. First, it is well defined whereas Model 4D is not. 
Second, the strong trait factors in Model 4E more accurately reflect 
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more parsimonious Model 4E nay be preferable. 
Summary of Analyses of Ostrgm data. 

None of the three parameterizations eliminated problems of poorly defined 
solutions for Ostrom data. The fixed factor loading parameterization was most 
prone to improper solutions. The Rindskopf parameterization was more likely to 
converge to proper solutions, but only at the expense of error /uniqueness 
estimates of zero with extremely large standard errors. In contrast solutions 
for method structure E *ere always well defined, suggesting that it eight be a 
more appropriate formulation of method effects. For the Ostrom data most of 
the variance can be explained in terms of either method factors or trait 
factors, whereas the inclusion of both trait and method factors produced onl/ 
a small improvement in fit. Since relatively little variance was uniquely due 
to either trait or method factors, any conclusions about their relative 
importance are problematic. Even more serious problems exist in the 
interpretation of the correlated method factors. These so-called method 
factors were more substantially correlated with an external validity criterion 
than were the trait factors, and apparently reflect trait variance instead of, 
or in addition to, method effects. The solution for Model 4E apparently 
provides a better representation of the MTMM data than the solution for Model 
40 selected by Widaman even though the fit of Model 4D is slightly better. The 
assumption of uncorrelated method effects in Model 4E is worrisome, but the 
traditional interpretation of the method factors in Model 40 is clearly 
unjustified and undermines any comparisons between it and other models. This 
illustrates the problems associated with using fit as the primary basis for 
selecting between alternative models instead of substantive interpretations of 
the parameter estimates. 

lYCQf and Shayelggn j[ 19861 Study. 
5escrietign of the Study and Data.. 

Byrne and Shavelson <1986) examined the relations between three academic 
self-concept traits <Math, Verbal, and School self -concepts) measured by 
three different self-concept instruments (M1-M3). School performance measures 
Nere also available for English and mathematics. Marsh and Shavelson (1985) 
reported Math and Verbal sel f -concepts to be nearly uncorrelated with each 
other even though both were substantially correlated with School self- 
concept. They posited two higher-order academic facets - verbal /academic and 
math/academic self-concept — to explain specific facets of academic self- 
concept. Their research posits a specific pattern of correlations among the 
trait factors and suggests that two general trait factors may provide a 
reasonable fit to the Byrne and Shavelson data. For the expanded MTMM models 
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more parsimonious Model 4E nay be preferable. 
Summary of Analyses of Ostrgm data. 

None of the three parameterizations eliminated problems of poorly defined 
solutions for Ostrom data. The fixed factor loading parameterization was most 
prone to improper solutions. The Rindskopf parameterization was more likely to 
converge to proper solutions, but only at the expense of error /uniqueness 
estimates of zero with extremely large standard errors. In contrast solutions 
for method structure E *ere always well defined, suggesting that it eight be a 
more appropriate formulation of method effects. For the Ostrom data most of 
the variance can be explained in terms of either method factors or trait 
factors, whereas the inclusion of both trait and method factors produced onl/ 
a small improvement in fit. Since relatively little variance was uniquely due 
to either trait or method factors, any conclusions about their relative 
importance are problematic. Even more serious problems exist in the 
interpretation of the correlated method factors. These so-called method 
factors were more substantially correlated with an external validity criterion 
than were the trait factors, and apparently reflect trait variance instead of, 
or in addition to, method effects. The solution for Model 4E apparently 
provides a better representation of the MTMM data than the solution for Model 
40 selected by Widaman even though the fit of Model 4D is slightly better. The 
assumption of uncorrelated method effects in Model 4E is worrisome, but the 
traditional interpretation of the method factors in Model 40 is clearly 
unjustified and undermines any comparisons between it and other models. This 
illustrates the problems associated with using fit as the primary basis for 
selecting between alternative models instead of substantive interpretations of 
the parameter estimates. 

lYCQf and Shayelggn j[ 19861 Study. 
5escrietign of the Study and Data.. 

Byrne and Shavelson <1986) examined the relations between three academic 
self-concept traits <Math, Verbal, and School self -concepts) measured by 
three different self-concept instruments (M1-M3). School performance measures 
Nere also available for English and mathematics. Marsh and Shavelson (1985) 
reported Math and Verbal sel f -concepts to be nearly uncorrelated with each 
other even though both were substantially correlated with School self- 
concept. They posited two higher-order academic facets - verbal /academic and 
math/academic self-concept — to explain specific facets of academic self- 
concept. Their research posits a specific pattern of correlations among the 
trait factors and suggests that two general trait factors may provide a 
reasonable fit to the Byrne and Shavelson data. For the expanded MTMM models 
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that include validity factors, each validity factor should be substantially 
correlated with the trait factors, particularly the trait factor in the 
•etching content area and, to a less extent, the school factor, and 
relatively uncorrected with the method factors. The Byrne and Shavelson 
study is unusual because there is jU st a good a priori basis for predicting 
the structure of the trait factors and also because two of the trait factors 
are relatively uncor related. 

My application of the Campbell -Fiske guidelines to the MTMH matrix 
(Marsh, in press) suggested strong support for convergent and discriminant 
validity; every convergent validity M as substantial, was larger than every 
heterotrait-heteromethod coefficient, and was larger than nearly every 
heterotrait-monomethod coefficient. For all three instruments School self- 
concept was moderately correlated with Math and Verbal self -concepts whereas 
Math and Verbal self-concepts were nearly uncorrected with each other. There 
mas evidence of some method effects associated with at least M3 and, perhaps, 
M2. The Byrne and Shavelson study is an exemplary HTMM study because of the 
clear support for the Campbell -Fiske guidelines, because of the large sample 
size <817, after case-wise deletion for missing data), because of the good 
psychometric properties of the measures, and because of the a priori knowledge 
of the trait factor structure. All models in Table 2 were fit ^.sing the fixed 
factor loading parameterization, the fixed factor variance parameterization 
was used for models that were poorly defined with the first parameterization, 
and the Rindskopf parameterization was used for solutions that were poorly 
defined by both standard parameterizations. 

Insert Table 6 About Here 
i§havigr of £he Solutions Undgr Different Pargmgigrizatigns A 

Nearly half of the solutions, 9 of 19, were poorly defined for the fixed 
factor loading parameterization (Table 6); 4 solutions were improper, and 5 of 
the solutions failed to converge. When these 9 poorly defined solutions were 
tested with the fixed factor variance parameterization, 2 of the solutions 
were well defined but the remaining 7 were still poorly defined. For the 
Rindskopf parameterization only 1 of the 7 problem solutions was improper but 
the all offending estimates were approximately zero and had large standard 
errors in the other 6 solutions. 

1. All four solutions for method structure E were well defined whereas 
all four corresponding models for method structure C were poorly defined. When 
the four solutions for method structure C converged to improper solutions with 
either of the standardized parameterizations, the parameter estimates for 
trait factors and the overall fit were the same as for the corresponding 
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solution for method structure E. For the Rindskopf parameterization, the 
parameter estimates varied somewhat, the fit was always somewhat poorer, and 
some parameters had values close to zero *ith large standard errors. For this 
application method structure E provides a better representation of method 
effects than method structure C. 

2. Model ID (correlated methods) produced the same improper solution, 
factor correlations greater than 1.0, for all three parameterizations. 
Because none of these parameterizations constrain factor correlations to be 
less than 1.0, they provide no protection from this problem. 

3. Model 3D converged to an improper solution with the fixed factor 
loading parameterization. Nhereas the solution mas constrained to be proper 
by the Rindskopf parameterization, the X mas approximately tmic. as large. 

4. For Model 4D the fixed factor loading parameterization resulted in an 
improper solution whereas the fixed factor variance parameterization produced 
a mell defined solution. The fit of the fixed factor variance solution mas 
someNhat poorer indicating that it apparently imposed a limitation on the 
solution space. 

Substantive Inter E retations of Jrait and Method Factors 

lQter E retatigns based on the MTMM datfc. Models positing only method 
factors fit the data poorly. Model 2B mith one so called general method factor 
and one general trait factor did substantially better. However, partly due to 
the may Model 2B mas specified, these two general factors represent the 
math/academic and verbal /academic factors that mere originally posited. The 
interpretation of either of these as a general method factor is unjustified. 

Model 2D (3 correlated M factors and 1 general T factor) provided a 
reasonable fit to the data, but inspection of the parameter estimates 
demonstrated interpreted onal problems. The three School measures should have 
loaded substantially on the general trait factor, but all three loadings mere 
small (.10 to .22). Loadings for Verbal and Math scales mere larger, but in 
the opposite direction, suggesting that the so-called general trait factor 
represented a bipolar (verbal vs. math self-concept) factor. For each of the 
•ethod factors all loadings mere substantial and positive, and the thre. 
•ethcd factors mere substantially intercorrelated (.80 to .97). However, these 
so-called method factors and the high correlations among them seem to 
represent convergence on general trait factor, associated mith each of the 
self-concept instruments. The interpretation of these factors is speculative, 
though substantively interesting, but the traditional interpretation of the 
factors is clearly unj ustifi ed. 

Insert Table 7 About Here~ 
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Model 4D, particularly given the large sample provide, a remarkably 

good fit to the data (TLI-.99). Parameter estimates (Table 6) indicate that 
each of the trait-factors is Nell defined. Consistent with theory and previous 
research, the School trait factor is substantially correlated with the Verbal 
and Math factors whereas the Verbal and Math factors are nearly uncorrected 
Hith each other. These trait factors are stronger than the method factors in 
that all nine measured variables have higher trait factor loadings than method 
factor loadings, but models without any method factors (e.g., Model 4A> 
provide a poorer fit to the data. 

Models 4C and 4E also provide good fits to the data (TLI=.977>. For 
Models 4C, 4D and 4E the factor loadings are similar for the School and Math 
traits, but the Verbal trait-factor loadings are smaller for Model 4D. 
Correlations among the trait factors are similar for all three models. Model 
4D may be preferred because it fits the data slightly better, but there is no 
compelling basis for rejecting the more parsimonious model 4E and the two 
Models lead to similar conclusions. 

Relations between MTMM factors and external validity criterion. Two 
validity factors defined by achievements in English and mathematics were added 
to selected MTMM models (Table 8). In contrast to the external validity 
criterion used with the Ostrom data, wethod effects associated with the self- 
report measures are unlikely to be related to the achievement test scores. 
Support for the validity of the interpretation of the MTMM solutions requires 
each achievement factor to be most highly correlated with its matching trait 
factor, less correlated with the School trait factor, substantially less 
correlated with the non-matching trait factor, and relatively uncor related 
with the method factors. The validity of the method factors would also be 
supported if the hypothesized pattern of correlations between trait factors 
and validity factors is improved by the addition of method factors. 

Five models contain three trait factors and two validity factors in 
combination with: correlated method factors (Model 40); uncorrelated method 
factors (Model 4C>; uncorrelated method effects represented as correlated 
errors (Model 4E); 1 General method factor (Model 48) \ or no method factors 
(Model 4A). For each of these models, there is reasonable support for the 
predicted pattern of correlations between validity and trait factors. The 
inclusion of method factors improves support for this hypothesized relation in 
terms of Verbal self -concept, but has little effect on predictions in relation 
to Hath self-concept or School self-concept. These results provide clear 
support for the a priori interpretation of the trait factors. 

The comparison of Models 4C and 4E is informative. Model 4C is ill- 
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defined but its TLI is much tetter than Model 4E. In Model 4C the validity 
factors are correlated with the method factors whereas in Model 4E no 
correlations were posited between validity factors and the error/uniquenesses 
that represent method effects. The results froe both M*fels 4C and 40 suggest 
that the method factors are correlated with validity factors, and this 
apparently explains the poorer fit of Model 4E. The hypothesized pattern of 
relations between trait and validity factors is also stronger for Models 4C 
than Model 4E. However, when selected error/uniquenesses in Model 4E Mere 
correlated with the validity factors (Model 4E' in Table 8) support for the 
posited pattern of correlations and goodness of fit were similar to Model 4C. 

Correlations between the trait and validity factors are similar in Models 
4C, 4D and 4E' (Table 81. The only substantive difference is that English 
achievement is somewhat more highly correlated with School self -concept than 
Verbal self-concept in Model 40, whereas English achievement is more highly 
correlated with Verbal self-concept than School self-concept for Models 4E' 
and 4C. Support for the posited pattern of relations is somewhat weaker for 
Model 40 even though its fit is best, Model 4C is poorly defined, and the 
correlations between method effects and validity factors are not easily 
represented in Model 4E'. There is no compelling reason for rejecting either 
Model 40 or 4E. In fact, the similar interpretations based on each of these 
different models suggests that the traditional interpretation of these models 
is probably justified. 

Insert Table 8 About Here 



Four models contain three method factors and two achievement factors in 
combination with: correlated trait factors (Model 40 discussed above), 
uncorrelated trait factors (Model 4C discussed above), 1 General trait factor 
(Model 20), and no trait factors (Model 10). Neither Models 20 nor 10 contain 
specific trait factors, and their method factors are substantially and 
positively correlated with the validity factors. I„ fact, English achievement 
is more highly correlated with M3 in Model 20, and mathematics achievement is 
•ore highly correlated with Ml and M3 in Model 10, than any trait factor, in 
any of the other models. These results provide clear support for the earlier 
supposition that these so-called method factor, contain substantial amounts of 
t-ait variance. In Model 2B there are just two general factors that might be 
interpreted to reflect a general method factor and a general trait factor. 
However, the correlations between these two general factors and the validity 
factors demonstrate that they reflect the math/academic and verbal /academic 
factors originally posited. 

fit iOf Analyses g£ ihj BzXQf iQ0 SbivtlKQ Mi* 
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The Byrne and Shavelson study is an exemplary MTMM study that should b. 
well suited to the CFA approach. For this reason it is particularly 
disappointing that so many of the eodels fro* Table 2 resulted in poorly 
defined solutions. So long as both correlated trait and either correlated or 
uncorrelated method factors were indudel in the MTMM model, support for the 
traditional interpretation of NTHH factors appeared reasonable. Homevw-, for 
models with no trait factors, or only one general trait factor, the 
interpretation of method factors as representing method variance was clearly 
unjustified. Instead, the so-called method factors reflected the influence of 
trait variance. Further support for this counter-interpretation was provided 
by the substantial correlations between these so-called method factors and 
the validity factors. 

Jhe Marsh and Irglgpd 11984) Study, 
Description gf £he Study and data. 

Marsh and Ireland (1984) asked each of six teachers (M1-M6) to evaluate 
139 student essays on six single-item scales of writing effectiveness: 
Mechanics, Sentence Structure, word Usage, Organization, Content /ideas, and 
Quality of style (T1-T6). Previous research reported a large general component 
of writing effectiveness suggesting that trait factors should be substantially 
correlated. There was no a priori hypothesis about the relative si*e of 
different trait correlations, but the traits were roughly ordered from lower- 
level components to higher-order components according to Foley's (1971) 
adaptation of the Bloom taxonomy. Unlike the first twj MTMM studies, ratings 
for the different methods were not completed by the same person. The teachers 
did not know any of the students who had written the essays or, typically, each 
other, and each teacher performed the rating task independently of the others. 3 
In addition to the 36 measured variables that constitute the MTMM data, a 
school performance measure of writing effectiveness was also available. My 
application of the Campbell -Fiske guidelines (Marsh t Ireland, 1984) suggested 
strong support for the convergent validity of all 6 traits. However, there was 
little support for discriminant validity and some indication of method effects 
associated with ratings by each of the six teachers. 
Behavior gf the Sglytigns.. 

Because the number of variables in this MTMM study was large only a 
subset of the MTMM models was tested with the fixed factor variance 
parameterization (Table 9). Nevertheless, 12 of these 13 models resulted in 
well defined solutions. Model 4B was improper in that several trait 
correlations exceeded 1.0, but this type of improper parameter estimate is 
unlikely to be eliminated by any of the parameter izat ions. Model 40 was 
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» 

technically i •proper in that the factor correlation Matrix (m Table 9) Mas 
not positive definite even though none of the correlations Mere greater than 
1.0. Despite these probleas, the solutions for the Marsh and Ireland data 
appear to be better b ehaved t han for either of the fir st two MTMM studies. 

Insert Tables 9 and 10 About Here 
SU&SiiQtiye. Inlficoreiaiion of JXiU §40 tStthaA Ei&lKft 

iQiKfirelftigQf bajgd. on £he BIBB g:§ii & The goodness-of-f it statistics 
(Table 9) demonstrated that such of the variance can be explained by either 
six correlated Method factors (Model ID) or six correlated trait factors 
(Model 4A>, but that Model ID fit the data slightly better than 4A. For Model 
4D aethod factor loadings Mere consistently much larger than trait factor 
loadings. A superficial inspection of these results Might suggest that the 
ratings reflect priaarily aethod effect, but there are probleas with this 
interpretation. First, it contradicts conclusions based on the Campbell -Fiske 
guidelines. Second, trait-factor loadings Mere substantially smaller in Model 
4D than 4A. This suggests that the so-called Method factors May represent 
general trait factors associated Mith each teacher and the high correlations 
represent agreement across teachers on this general trait. 

BBglUUgaS between MTMM factor? and. the validity factor. In order to 
test the counter interpretation of the aethod factors, the school perforMance 
Measure Mas added to Model 4D. The parameters for the MTMM variables Mere 
relatively unaffected by the inclusion of this additional variable. However, 
the school perforMance factor Mas substantially More correlated Mith the so- 
called aethod factors ( .56 to .68) than Mith the trait factors ( .14 to .30). 
Because this pattern of results is so implausible, the traditional 
interpretation of the so-called Method factors in this Model Must be rejected. 
Models 2D and ID also posited correlated Method factors and correlations 
between these method factors and the validity factor Mere also very high ( .33 
to .75), whereas the general trait factor in Model 2D Mas only modestly 
correlated with the validity factor. In contrast to Models with correlated 
methods, Model 4C posited method factors to be uncorrelated. For Model 4C 
correlations between the trait factors and the validity factor varied betMeen 

.71 and .BV whereas 5 of the 6 correlations betMeen method factors and the 
validity factor Mere nonsignificant. Even though the fit for Model 4C Mas 
somewhat poorer than Model 4D, the substantive interpretation of the solution 
argues that it better reflected the MTMM data. 
Bttibgl Itrycture g L 

The first two MTMM studies both contained three trait factors, and for 
such MTMM studies method structure E results are equivalent to those for 
method structure C so long as the solutions are well defined. However, when 
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there .re .or. than 3 trait,, a. here, th. two structure ar. not equivalent. 
For T » 6, -ethod structure C use, 6 parameter, to define .ach method factor, 
whereas there are IS (T x (T-D/2) correlation, *»ong the error /uniqu«>e..e. 
associated wit h each Method. Insofar as -ethod structures C and E are both 
well defined and fit the data equally well, then the more parsiaoniou. .ethod 
structure C is preferable. Hoover, th. fit of Models based on .ethod 
structure E .as Mich better than those based on .ethod structure C for this 
data. This suggests that the 15 correlated err or /uniquenesses associated with 
each aethod effect cannot be explained by a single .ethod factor and that 
aethod effects do not have a congeneric-like structure. This is very important 
in that all the aethod factor, in the entire taxonoay are based on these 
assumptions. This also explains why Model 4E (TLI-.948) fits the data better 
than Model 40 (TLI-.935) even though Model 4E posits unc or related aethod 
effects whereas Model 40 posits correlated aethod factors. 

The superiority of Model 4E over 4C is also shown in the expanded aodels 
containing the validity factor. Correlations between trait factors and the 
validity factor are substantial for both Models 4E and 4C, but are higher for 
Model 4E. As noted for the first two MTMM studies, correlations between aethod 
effects and the validity factor are not easily incorporated into Model 4E. 
However, correlations between validity and aethod factors were saall and 
generally nonsignificant for Model 4C. Similarly, inspection of the 
edification indices provided by LISREL (see Joreskog & Sorboa, 1981) 
indicated that error /uniquenesses in Model 4E were essentially unc or related 
with the validity factor. For this reason no alternative Model corresponding 
to Model E' in the first two studies was proposed. 

SUMMARY AND IMPLICATIONS 
Why does one use the kinds of analyses discussed here? One perception in 
accordance with my own is that the activation for MTMM analyses has been the 
desire to establish specific trait representations in Measures. Method 
variance is seen as contaainating that representation. The CPA approach as 
traditionally applied has Modelled trait and aethod factor, as if they were 
equally important. The approach advocated here places greater emphasis on 
the interpretation of trait representations. This is accomplished by 
comparing different aodels to determine if the introduction of method factors 
substantially alters the interpretation of trait representations, by 
introducing an alternative method structure (aethod structure E) that 
apparently provides a more accurate representation of the trait 
representation, and by demonstrating how external validity criteria can be 
used to test the validity of the traditional interpretations of different 
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models. Whereas the conventional approach was due at least in part to early 
work by Joreskog, the perspective taken here is consistent with Joreskog's 
statement that "method factors are what is left over after all trait factors 
have been eliminated" (1971, p. 128). 

Despite the growing enthusiasm for the CFA approach to MTMH data, 
problems demonstrated here call into question its value, the traditional 
interpretation of MTMH factors, and the validity of previous MTMH research. 
The most important of these problems are the technical difficulties in 
estimating parameters and the interpretation of so-called method effects that 
apparently represent the effects of trait variance in addition to, or instead 
of, method variance. So long as problems as basic as these remain unresolved, 
the promise of the CFA approach to HTHH data cannot fulfilled. 

The CFA approach to HTHH data is plagued by technical difficulties in the 
estimation of parameter values and different parameter i rations of HTHH .odels 
were propossd to eliminate such problems. The fixed factor loading 
parameterization was apparently most prone to ill-defined solutions, whereas 
the Rindskopf parameterization was most likely to converge to proper 
solutions. However, when error /uniquenesses were negative for the standard 
parameterizations, the offending parameters were estimated to be close to zero 
(also see Dillon et al., 1987) with very large standard errors with the 
Rindskopf parameterization. Hence the Rindskopf parameterization did not solve 
the the problem, but merely made the manifest symptoms less obvious. 

Method structures in Hidaaan's 1985 taxonomy and those used in most 
applications of CFA to HTHH data posit a separate method factor associated 
with each method of assessment. An alternative conceptualization, Method 
structure E, was formulated in which method effects arc represented as 
correlated error /uniquenesses. Method structure E has three important 
advantages over eethod structures C and D. First, models with method 
structures C and D were frequently ill-defined no matter what parameterization 
was used, whereas models based on method structure E were always well defined 
in the present applications. Second, when there were more than three traits, 
method structure E provided a test of the implicit assumption that all the ' 
correlated error /uniquenesses associated with a single method of assessment 
could be explained in terms of a single method factor. The importance of this 
second advantage was demonstrated for the Harsh and Ireland data in that Model 
4E provided a better fit than the corresponding Model. 4C and 40. Third, 
•mthod structure E apparently provided a more accurate interpretation of trait 
variance than alternative models when these interpretations were evaluated in 
relation to external validity criterion. In this respect, the use of external 
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validity criteria to validate interpretations of the method and trait effects 
is an important conceptual innovation. 

The most serious potential problem with hTMM models is the implicit 
assumption that so-called method factors represent primarily the effects of 
method variance. If this assumption is violated, then the interpretation of 
trait and method factors in most CFA studies and the detailed comparison of 
nested models proposed by Midaman (1985) may be unjustified. Results from 
the MTMM studies considered here suggested that this assumption is often 
implausible. In all three MTMM studies the so-called method factors for at 
least some of the MTMM models apparently represented trait variance in 
addition to or instead of method variance (also see Marsh fc Butler, 1984, for 
another compelling example). When there actually are distinct traits that are 
at least moderately correlated, this phenomenon is most likely in models that 
posit correlated method factors (method structure D). Using method structure 
D the problem is likely to be most severe in models that posit no trait 
factors (ID) and to become less severe as the trait structure proceeds from 1 
to 4. The problem will apparently be least likely when method factors Mere 
required to be uncorrelated as in method structures C and E. 

The emphasis of the present investigation has been on potential problems 
in the interpretation of so-called method factors that really reflect variance 
that should be attributed to a general trait effect. This is consistent with 
Joreskog's (1971) conceptualization of method effects as what is left after 
trait factors have been removed and my perspective on the the intent of MTMM 
analyses. It is important to note, however, that the converse phenomenon may 
also exist. That is, it is possible that so-called trait effects really 
reflect variance that should be attributed to a general method effect. If an 
appropriate method structure is not employed, then so-called trait factors may 
represent method variance in addition to, or instead of, trait variance. An 
unresolved conceptual and technical problem is how to discriminate between 
method and trait factors when both ere highly correlated. In the extreme, it 
is easy to imagine the case where a MTMM matrix of correlations produced by 
highly correlated trait and method factors could be explained by a single 
factor (Model 1B/2A). Whereas this situation would clearly indicate a lack 
discriminant validity there would be little basis for determining whether the 
single factor represented a general trait effect, a general method effect, or 
a combination of the two. 

The taxonomy of MTMM models in Table 2 was based in large part on 
Widaman's Uxonomy. Midaman (1983) used essentially the same CPA approach, 
•any* of the same MTMM models, and even analyzed one of the same MTMM studies. 
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Because Hi daman's evaluation of th» re a a „„,„„k 

u«ion ot the CFA approach was much more optimistic than 

•me, it is informative to critically evaluate his findings in relation to the 
criteria used here. widaman did not provide a detailed report of the behavior 
of his CPA solutions, but results reported here indicate that poorly defined 
solutions occurred for the Ostrom data considered in both studies. Widaman 
chose to present five MTMM solutions as the most appropriate representations 
of his MTMM analyses. However, fur of these solutions had error /uniquenesses 
of zero in conjunction with large standard errors whereas the fifth solution 
required a correlation between two method factors to be 1.0. Hothke (1984) 
also reported that 21 MTMM matrices - including the three analyzed by 
Widaman - resulted in poorly defined solutions when he fit Model 40. 
Apparently, none of the solutions chosen by eidaman was well defined according 
to criteria used here suggesting that widaman was also plagued by poorly 
defined solutions, widaman did not report a critical valuation of alternative 
interpretations of hi. method factors, but results reported here suggest that 
this was a problem for the Ostrom data. Using criteria described earlier there 
is reason to suspect that so-called method factors in at least some of 
eidaman's results of other MTMM matrices may have also represented trait 
variance in addition to, or instead of, method variances. In summary, a 
critical evaluation of Wdaman's results provides little basis for optimism 
about the application of CFA to MTMM data. His results suggest the same sort 
of problems that were identified here. 

RECOMMENDATIONS 

Problems with the CPA approach to MTMM data appear to be most serious 
for MTMM studies in which method effects are substantially correlated and for 
MTMM models that posit correlated method factors. Campbell and Fiske (1959) 
originally stressed that the multiple methods should be as distinct as 
Possible, and this advice seems appropriate for the CFA studies as well. The 
choice of method effects is, however, often dictated by the nature of the 
study, and the pattern of correlations among method factors may be difficult 
to determine a priori. Particularly when both traits and methods are 
substantially correlated, the researcher must critically evaluate the MTMM 
solutions for alternative interpretations. Because the traditional 
interpretation of trait and method factors may frequently be unjustified, the 
burden of proof lies with the researchers to demonstrate that they ar. 
justified. This requires that more emphasis be given to the substantive 
interpretations than ha. typically been the case in CFA studies. 

The use of uncorrected traits may also be helpful, though this is 
unusual in MTMM studies. Byrne and Shavelson (1986), however, did consider 
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and method factors (4D and 4C> and method effects represented as correlated 
uniquenesses (4E) seem most useful, supplemented perhaps by those positing 1 
general factor (1A/2B), only trait factors (4A>, and only method factors 
(ID). Particularly when Models 4D and 4E are both well-defined and lead to 
simiUr conclusions as with the Byrne and Shavelson (1986) data, then Me 
traditional interpretation of these models is probably justified. In this 
case it may be reasonable to base inferences about method and trait effects 
on just these models — dispensing with other models altogether. Other 
models from the taxonomy or models idiosyncratic to particular substantive 
issues may, however, provide useful supplemental informatior, about the data. 
Because model 4E has not been widely applied elsewhere, it is important to 
further examine its apparent advantages in other studies. A particularly 
useful evaluation would be to apply various models — including method 
structure E — to simulated data in which the underlying factor structure 
was known. Subject to the results of this further research I recommend that 
Model 4E should be at least one of the MTMM models examined in all 
applications of CPA to MTMM data. 
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Footnotes 

1 - Campbell and Fiske (1959, p. 85) stated that "the presence of method 
variance is indicated by the difference in level of correlations between 
parallel values of the monomethod block and the heteromethod block, 
assuming comparable reliabilities among the tests." Marsh (in press) 
operational ized this statement to provide estimates of the relative size of 
method effects associated Hith each method of assessment and discuss-d 
limitations in the inferences based upon it. 

2 — Standard errors of estimated parameters that were extremely large were 
indicated to be 1.0 by Nidaman (1985), though the footnote indicating this 
was mistakenly omitted from the published article (Widaman, personal 
communication, 3 September, 1987). 

3 ~ The CFA aporoach to MTMM data assumes that the different methods 
represent fixed effects. Whereas this limitation may be reasonable for 
some aoplications, it is probably inappropriate for the Marsh and Ireland 
data where the different raters more realistically constitute a random 
effects facet (i.e., a sample of potentially much larger sample of raters). 
I am not aware, however, of any solution to this problem. 

4 — This perspective was expressed by an anonymous reviewer. 
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Table I 

Ngwtort To It Eatleatod Far the tooeral HTHH Hodel I Hodel «) 
Factor Loodiofs (laabda Y) Error/Uaiweeesses (Thtta) 
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jgtf t Coat* irutory factor analysis ICFA) eodels to It considered ia this 
iamtifatioa art defined in terns of tkt throe L1SKL desife oatricM 
presented here. The HTM problee sheen here it haa 3 traita factors (Tl - T3) 
and 3 eetbod factara (HI * R3) that are defied ia teres of the f oeasored 
variable (tlol - t3o3). Ml Nraoeters eitb vale * of 0 or I are fiied aad 
not estioated ehereae all other pareoetera are estiaated oitheot caaatraiat. 
The pritTterizatioo shone here, eith facto variaaces lie PSD fixed to be 1, 
ia refrred to aa the f tied factor variaace Mraoeterization. 
a 

For too fiied factor loediee. earaeeterizatioo tbooo factor loading* ooold bo 
fiiod to bo I and factor variances ootid bo freed. For the fiied factor 

toadies, earaeeterizatioo theoc factor variaaces ooold be fixed to be I and aa 

c 

factor loadioes Mold be fiied to I. For eetbod stroctore E (tec Table 2) 
tkeoo correlation bet woo error/ooiweaesies ooold be estioated, and the 
eetbod factor* and their associated eeraaeters ooold be elioiaated froa Laabda 
VaadPSl. 



0 

ERJC 



32 



Mtitrait-MltiMthoi data 

Till* 2 

TaMMg *f ttmctral IMtU fir WW lata MaatH Fraa Hdaaaa IMgl 
totted Stmctart 
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General factors are define* to be oncorrelate* oith other factors in the oo*el. Mthoegh 
feneral factors are posite* to represent either trait variaoce or oetbo* variance, tbis 
assooptioo Mill oot alnays be accorate an* oay be difficult to test. Ho*els 2* an* II are 
equivalent, an* it is federally tot possible to deter nine ehetber the ooe feneral factor 
reflects trait variaoce, oetbo* variaoce or soot coobioatioo of trait ao* oetbo* variance. 
Ho*el 21 retires additional coostraiots tbat oay be arbitrary ao* tbat oay oot provide 
equivalent solutions. Hetce its osefoloess oay be *nbioas no less there is ao a priori basis for 
the constraints. Hodels voder oetbod stroctnre E have no eetbod factors. Instead oetho* 
effects are inferre* on the basis of correlate* error /uniquenesses (see Table 1). This oetho* 
structure, particularly nben there are three traits, corresponds oost closely to oetho* 
structure C in nhich there are H encor relate* oetbods. 
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Table 3 

Suaaary of Soodnats of Fit end Solution lehavior lor tho Ostroa Data 
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HStti All aodalc Mara first taatad with both tha f txed factor loading and tht 4 ixed factor 
variance peraaeterlzatlons. However, in all casas in which tha fixed factor loading 
peraeeterizatlon rasultad in a Mall deflnad solution tha solution for tha fixed factor variance 
peraeeterizatlon Mas tha saaa and so it is not presented. When neither of these standard 
paraaeterliatlons resulted in a well-defined solution the Rlndskopf pereaeterliatlon Mas used. 
For aodels that also had a validity factor, only the Rindskopf paraaeterlxatlon Mas used 
because the previous analyses showed that this parameter it at ion facilitated convergence for 



this application. 

see Table 2 for a description of the Models. 



b c 

TLI • Tucker Lewis Index. Probleasi 1 • 



failed to converge in 250 iterations! 2 - factor correlation > l.Of 3 ■ Negative factor 
variance) 4 - Negative error /unlquenessj 5 - estlaates with excessively large standard errors. 
Problees 2, 3 and 4 ware only cumin* if the solution converged, and problems Mas exaeined 
only if tha solution converged and had no out-of -range paraaeter estlaates. Validity factor 

used to represent aathod 

34 



RIO* corr * ut * tf " lth tralt ** ctor * M f* 0 * «<th ony error /unl 
==jlrteU in Hotel 4C MharoM two ouch correlation* opp«§rijLl!WrJ-§f! 



MUUVUtttlllMOM MU 

Mill 

hrmtm Estiaetee Far MU « Far tat Mm tola 



FactrlMHn Errv/MfMMMN CIMtol 



varieties 


Tl 12 

f * 1* 




till t2al Utl tie? ttt tW tli3 t2i3 t3a3 tla4 t2a4 t2M 


till 


.IN • 


a 

w 


.341 






t2al 


0 .741 


A 

V 


.11 .141 






till 


• 0 


.111 

• ■If 


-.11 .94 .Ml 






tU2 


.I7t • 


9 


• • • 


.231 




ttt 


• .fM 


9 


• • 9 


.931 .29 




tie? 


• • 


.M 


• • 0 


.94 .941 .221 




tias 


.411 0 


9 


0 9 1 




.911 


oa 


• .741 


• 


• • 1 




-.03 .42 


tw 


• • 


.fM 


III 




.11 .01 .341 


tU4 


.Ml • 


• 


• I I 


\ 


•01 .211 


t2e4 


• .711 


• 


• I I 




0 0 0 .lit .441 


t3e4 


• • 


.741 


III 




0 I 9 .141 .131 .431 



Fact* Variaacf/CovariMCff (PSD 
Factfrt Tl T2 TJ 

Tl 1 

T2 .Ml 1 

T3 .Ht.f4t 1 
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Tabic 5 

Correlations Between Trait (T1-T3), Method (M1-M4), General (61-62) and 
Validity (VI) Factors for Selected MTMM Models Based on Ostro* study 
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MTMM Factors 



Models Tl T2 T3 Ml M2 M3 M4 61 G2 VI 
Model 4E' 

Tl 1 
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T2 . 96* 1 

T3 .97* .94* 1 

61 0 0 0 1 

Vi .69* .76* .58* -.21* 1 

Model 4A 

Tl 1 

T2 .98* 1 

T3 .99* .97* 1 

VI .67* .76* .60* 1 

Model 2D 

Ml 1 

M2 .97» i 

M3 .96* .95* 1 

g -— .88* .87* .88* . 

VI .59* .62* .59* .51* --- !s9* 1 

c 

Model ID 

Ml 1 

M2 .97* 1 

M3 1.01* .99* 1 

M4 .89* .87* .93* 1 

VI .70* .68* .73* .58* 1 



ERIC 



Ngtej. Only parameter estimates from the matrix of factor correlations (PSD 

are presented. 

t p < .05 
a 

Tl * Trait 1, T2 * Trait 2, T3= Trait 3, hl= Method 1, M2«Method 2, H3« 

Method 3, Gl « General 1 (general method) 9 G2 * General 2 (general trait), VI 

b c 
■ Validity Criterion 1. See Table 2 for a description of the models. This 

model resulted in an improper solution in that 1 of the method factor 

correlations Mas greater than 1.0. Similar results were found when the 

validity criterion was not included (see Table 3). 
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off Boodw— ■ off Pit and Solution fahavior ffor tha tymo and flhavolaon Data 



Paraaatorlzatlon 



Fftnad Factor Loadinga Finad Factor Variance Rlndokopff Faraoatariaatlon 



Nodal* 


2 

X 


dff 


2 

X /dff 
Ratio 


b C 
TLI Problea 


2 

X 


dff 


2 

X /dff 
Ratio 


TLI ProblM 


2 

X 


df 


2 

X /df 
Ratio 


TLI Pr ob 1m 


Without Validity Criteria 






















1A 


5272 


36 146.44 


.000 























1B/2A 


2363 


27 


87.99 


.403 























1C 


3807 


27 


141.02 


.037 


1 


3807 


27 


141.02 


.037 


1 


3934 


27 


145.71 


.005 5 


ID 


2302 


24 


93.92 


.347 


2 


2302 


24 


93.92 


.347 


2 


2302 


24 


95.92 


•347 2 


IE 


3807 


27 




.823 























28 


S34 


20 


26.70 























2C 


1698 


18 


94.34 


.358 


1 


1528 


18 


84.88 


.423 


4 


1528 


18 


84.90 


•423 5 


20 


383 


IS 


25.63 


.831 


1 


323 


IS 


21.52 


.839 













2E 


1328 


18 


84.90 


.423 























3A 


1107 


27 


41.01 


.726 























38 


313 


18 


17.37 


.887 


1 


310 


18 


17.21 


.889 


1 


387 


18 


21.51 


ma m 

• 83V 9 


3C 


708 


18 


39.31 


.737 


1 


707 


18 


39.26 


.737 


4 


707 


18 


59.28 


,737 5 


30 


94 


IS 


6.30 


.964 


3 


135 


IS 


9.03 


.943 


1 


188 


IS 


12.54 


•921 5 


3E 


707 


18 


39.26 


.737 























4A 


431 


24 


18.81 


.878 























48 


112 


IS 


7.49 


.953 


3 




















4C 


63 


IS 


4.31 


.977 


3 4 


65 


IS 


4.31 


.977 


1 


66 


a as 


4. 38 


a ft it 


40 


28 


12 


2.33 


.991 


1 


40 


12 


3.32 


.984 













4E 


63 


12 


4.31 


.977 























With Validity Criteria 
























1A 












6399 


45 


142.21 


.000 













10 












2699 


38 


71.03 


.304 


4 










28 












800 


36 


22.23 


.830 













20 












469 


27 


17.35 


.884 













4A 












697 


38 


18.33 


.877 












4B 












268 


27 


9.92 


.937 












4C 












132 


27 


3.73 


.967 


1 


148 


23 


6.45 


.961 5 


40 












73 


20 


3.63 


.981 












c 

4E 












363 


29 


12.32 


.918 












c 
4E' 












149 


26 


3.73 


.966 













H8tf A All models were first tnttd with thu fixed ♦actor variance parameterization, models 
••ith solutions that had problems were than tested with tha fixed factor variance 
parameterization, and if thara were atill problems with tha Rindskopf parameterization. Tha 
axpandad aerial • with tha validity criteria Mara tested with tha fixed factor varianca 
parameterization and than tha Rindskopf parameterization was usad if a poorly daf inad 



solution was obtained. 

a b c 

aaa labia 2 for a description of tha moaels. TLI » Tuckar Lawis Index. Problems 1 « 

failed to converge in 250 iterationsi 2 - factor correlations > 1.0| 3 - Negative factor 

variance | 4 ■ Negative error /uniquenessi 5 ■ estimates with excessively large standard 

errors. Problem 2, 3 and 4 were only examined if the solution cbn verged, and problea 5 was 

O examined only if the solution converged h&d no out -of -range parameter estimates. In 

E|yCiodel 4E f error /uniquenesses used to represent method effects were uncorrected with the 

validity factor whereti for Model 4E f selected correlations were estimated. 
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Hultitrait-ooltioethod data 



Taklt 7 

Parameters For Hodel 41 For too lyree ood Shavelson lata 



Variables 


Foctor loediais (Laoodo Y) 

Tl T2 T3 HI H2 H3 


Error/ 
Itnifnei 


tlol 


.841 0 


0 .18 


0 0 


.271 


t2ol 


0 .571 


0 .561 


0 0 


.351 


t3ol 


0 0 


.941 .03 


0 0 


.111 


tlo2 


.Ml 0 


0 0 


.264 0 


.481 


t2o2 


0 .701 


0 0 


.651 0 


b 

.07 


t3o2 


0 0 


.931 0 


.194 0 


.121 


tlo3 


.771 0 


0 0 


0 .601 

ft 


b 

.03 


t2o3 


0 .871 


0 0 


0 .331 


.151 


t3o3 


0 0 


.851 0 


0 .251 


.181 



Factor Variaoce/Covariances (PSD 

Factor* Tl T2 T3 HI H2 H3 

Tl 1 

T2 .591 1 

T3 .601 .04 1 

HI 0 0 0 1 

82 0 0 0 .801 1 

R3 0 0 0 . 25 .22 1 



Hole. The throe trait* arc School self-concept (Tl), Verbal self-concept 
(T2) aod Hath self-concept (T31 oherea* the three oethods corresponding to 
three different self-report instroeeots used to eeasere each of these 
facets of self (R1-H3). Ml parameters nth values of 0 or 1 are fiied and not 
estioated ohereas all other paraoeters arc Mtioated oithoot coostraiot. 
The fiied factor variaoco paraoetcrization, aith factor variances (in PSD 
fiied to be 1, nas osod to estioate parameters, 
a 

lecaose error/uniquenesses were constrained to be oncorrolated in this 

oodel the estioatcs are presented in this fori to save space (see Theta 
b 

oatrii in Table D. Mtboofb these error /uniquenesses did not differ 
significantly froo zero, their standard errors were very soall. 
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TaMa 8 

Correlations BetMsan Trait (Tl-TS), Hethod <M-H3>, Gmral <61-C2) and 
Criterion (C1-C2) Factors for Selected mm hotels iased on tyrne and 
Shavelson Study 



b 



a 

NTNN Factors 



Hodels Tl T2 T3 Hi H2 MS 61 62 VI V2 

Nodal 4E' 



Tl 1 

J2 .611 1 

T3 .61$ .09 1 

VI .961 .631 .211 — 1 

V2 .481 .071 .97$ .52$ 1 

nodal 4€ 

Tl 1 

T2 .61$ I 

T3 .61$ .09 1 

VI .94$ .43$ .22$ 1 

V2 .46$ .08$ .99$ .32$ 1 

Hods! 40 

Tl 1 

T2 .61$ 1 

T3 .61$ .06 1 

m o o o .i 

H2 0 0 0 .89$ 1 

fP °-_ 0 0 .22 .18 1 

VI .99$ .97$ .21$ -.13$ -.16$ .10$ 1 

V2 .41$ .06 .96$ .07 .02 .37$ .92$ 1 

Hodal 4C 

Tl 1 

T2 .62$ 1 

T3 .61$ .06 1 

HI 0 0 0 1 

rg 0 0 0 .89$ 1 

«3 0 0 0 .22 .18 1 

VI .92$ .60$ .21$ -.29$ -.38$ .21$ — 1 

V2 .40$ .07 .97$ .07 .02 .31$ .52$ 1 

nodal 4B 

Tl 1 

T2 .66$ 1 

T3 .63$ .10$ 1 

61 0 0 0 1 

Vi .34$ .90$ .20$ -.32$ 1 

V2 .48$ .12$ .99$ -.29$ .92$ 1 

nodal 4A 

Ik 1 

T2 .66$ 1 

T3 .63$ .08$ 1 

VI .58$ .44$ .22$ 1 

V2 . 90$ . 09$ . 60$ . 52$ 1 

nodal 2B 

Vi — HI ?53 $ | 2 1$ 1 

V2 — .11$ .61$ .52$ 1 

nodal 20 

HI 1 



s 



.98$ 1 

.89$ .82$ 1 

62 0 0 0 1 

VI .54$ .39$ .62$ .08 1 

V2 .38$ .30$ .48$ -.47$ .52$ 

Hodal 10 C 

HI 1 

J12 1.11$ 1 

M3 1.05$ .95$ 1 

Vi .36$ .22$ .35$ — 1 

V2 — - .66$ .94$ .70$ .52$ 



Mitt Only paraaatar astiaatas fro* tha aatrix o4 factor conflations (PSD 
are prasantad. 

$ p < .05 

s b 
Sat Table 4 for a dascription of tha HTHH factors. Saa Tabla 2 for a 

c 

dascription of tha aodals. This aodal rasultad in an improper solution 



*lt* 2 of 3 aathod f attar corralatlans baing §r9&tmr than 1.0. q q 



Hultitrait-eultiaethod data 

mitt 

Sueeary of feodum of Fit and Solution lokavior for the Harsh and Ireland 
lata 

a 2 1 /df b c 
Nndol 1 df Ratio TLI Probleo 



MtfcOHt Validity uitoria 



in 


6407 430 


10.17 


.000 


— 


1I/2A 


2439 993 


4.11 


.660 


— 


1C 


1V91 394 


3.36 


.742 


— 


10 


1296 379 


2.24 


.065 


— 


IE 


1390 340 


2.94 


.700 


— 


2C 


1199 330 


2.15 


.075 


— 


2D 


997 343 


1.04 


.909 


— 


2£ 


032 304 


1.69 


.925 


— 


4A 


??94 479 


3.96 


.677 


— 


4B 


1776 343 


3.27 


.752 


2 


4C 


901 343 


1.01 


.912 


— 


4D 


043 320 


1.60 


.935 


— 


4E 


722 409 


1.40 


.940 


— 


Kith Validity Criterion 






1C 


6593 666 


9.90 


.000 


— 


ID 


1334 610 


2.19 


.067 




20 


1033 373 


1.03 


.909 




2E 


096 540 


1.66 


.923 




4A 


2331 610 


3.02 


.603 




40 


1014 573 


3.17 


.757 


2 


4C 


1009 360 


1.70 


.913 




40 


073 333 


1.64 


.920 




4E 


730 520 


1.56 


.940 





d 



Mole. Oil oodeli nere first tested with the fiied factor variance 
paraoeterization. 

n b 
we Table 2 for a description of the oodels. TLI ■ Tucker Levis Indei. 

c 

Probleosi 1 ■ failed to converge in 250 iterations; 2 ■ factor c 
correlations > l.Oj 3 » Relative factor variance; 4 ■ Negative 
error/unioueness; 3 » estioates uith eicessively large standard errors of 
Htioate. Problem 2, 3 and 4 acre only eianined if the solution 
converged, and probloo 3 oas essoined only if the solution converged and had 
no out-of-range paraoeter estioates. 
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Hultitrait-auitiaethod data 

Talle 10 

Correlation* OetNeea Trait (T1-T6), Method (Hi-Hi), 6eneral (61-621 and 
Validity (VI) Factor* for Stitctod HTHH Model* laud on Karon and Ireland Data 



Factor* in HTHH Hodel* 



Tl T2 T3 T4 T3 T6 HI H2 10 H4 HS H6 61 62 VI 
Hodel 4E 
Tl 1 

T2 981 1 

T3 92$ 96$ 1 

T4 931 941 931 1 

T3 901 911 931 961 1 

T6 941 961 931 98$ 981 1 

VI 811 801 771 831 761 801 1 



Hodel 4D 

Tl 1 

T2 991 1 

T3 641 761 1 

T4 701 741 611 1 

T5 571 581 621 841 1 

T6 781 791 681 931 981 1 

HI 0 0 0 0 0 0 1 

H2 0 0 0 0 0 0 711 1 

H3 0 0 0 0 0 0 721 711 1 

H4 0 0 0 0 0 0 721 731 791 1 

5 J { 0 0 0 0 661 631 791 771 1 

Sf L L. ?- 0 0 0 721 721 781 671 > 

VI 281 231 14 301 17 221 661 561 691 681 661 671 1 

Hodel 4C 

Tl 1 

T2 991 1 
P 921 931 1 
T4 921 941 921 1 

I? 5?! S 1 w '7t 1 
T6 941 951 941 98$ 99$ 1 

HI 0 0 0 0 0 0 1 

H20000000 1 

H300000000 1 

14000000000 1 

1*30000000000 1 

"6000000000001 

VI 77$ 75$ 71$ 79$ 73$ 75$ 08 -10 06 12 08 16$ 1 

Hodel 4E 

Tl 1 

T2 98$ 1 

T3 92$ 96$ 1 

T4 93$ 94$ 93$ 1 

T5 90$ 91$ 93$ 96$ 1 

T6 94$ 96$ 93$ 98$ 98$ 1 

VI 81$ 80$ 77$ 83$ 76$ 80$ l 



Hodel 48 
Tl 1 

T2 106$ 1 
T3 64$ 80$ 1 

T4 66$ 84$ 77$ 1 

T5 45$ 59$ 101$ 100$ 1 

T6 71$ 82$ 81$ 110$ 110$ 1 

61 0 0 0 0 0 0 i 

VI 27$ 26$ 20$ 35$ 26$ 29$ 74t i 

Table 10 continued on next page 
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Taele 10 Continued 



Hultitrait-oultiaethod data 



Factor! in HTHH Hodels 



Tl T2 T3 T4 T3 T6 HI 02 H3 M ffi Hi 61 62 VI 
Hotel 4ft 

Tl 1 

T2 1041 1 

H 961 991 1 

T4 961 981 981 1 

TS 941 951 1011 1031 1 

T6 911 991 1001 1041 1071 1 

VI 111 801 761 831 771 801 i 



Hotel 20 

HI 1 

E 738 1 

S 711 701 1 

2J 741 771 791 1 

16 671 691 781 781 1 

s p p p rn , 

V1 651 531 671 671 631 64! 330 1 
Model ID 

HI 1 

5 751 751 1 

25 781 791 821 1 

2? 731 731 811 80! 1 

2* 77! 76! 81! 721 70! 1 

WI 731 61! 75$ 741 711 721 721 1 



Note Only factor correlation froo the Pii eatrii are presented. Parameter 
eotiiatcf are presented nithout tecioal points, ftll paraaeters eith values of 
1 or 0 are fiied, idtereas factor correlations greater than 100 are oot-of- 
range estioates. 
I f < 05 

See Tahle 4 for a description of the HTHH factors. See Table 2 for a 
description of the aodels 
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